Skip to content

hack/ci: do not swallow apiv2 failures - #29528

Open
vtushar06 wants to merge 1 commit into
podman-container-tools:mainfrom
vtushar06:fix-apiv2-exit-status
Open

hack/ci: do not swallow apiv2 failures#29528
vtushar06 wants to merge 1 commit into
podman-container-tools:mainfrom
vtushar06:fix-apiv2-exit-status

Conversation

@vtushar06

@vtushar06 vtushar06 commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

The apiv2 bash suite has not gated CI since 66c6da6. run_apiv2 puts both make targets inside one sh -c "( ... )" with no set -e, so only the last command decides the exit status and make localapiv2-bash is discarded. set -eo pipefail at the top of runner.sh is the outer bash and does not reach into that inner sh.

It is failing right now on main. Run 31847250452, job apiv2 rootless fedora-current, conclusion success, log line 3089:

make: *** [Makefile:762: localapiv2-bash] Error 17

Same thing in the three earlier green main runs I checked (31687097967, 31540756565, 31514570842), each with plan 1..2191 and Error 17.

I used an accumulator rather than set -e so the python half still runs when the bash half fails - otherwise we would lose that half of the report on every failure. Checked the shapes on bash 5.2: the current one exits 0 when the first target fails, this one exits 1 and still runs both.

I have not looked into the 17 failures themselves yet, they look like three clusters in 26-containersWait and 27-containersEvents. Happy to open a separate issue for those once this gates again.

@vtushar06
hack/ci: do not swallow apiv2 failures
0f12269 
Signed-off-by: Tushar Verma <tusharmyself06@gmail.com>
@github-actions github-actions Bot added the CI label Aug 15, 2026
@Luap99

Luap99 commented Aug 17, 2026

Copy link
Copy Markdown
Member 
not ok 1347 [26-containersWait] POST containers/WaitTestingCtr/wait?condition=next-exit [-d {}] : status
#  expected: 200
#    actual: 500
#  response: {"cause":"unable to open a handle to the library","message":"unable to open a handle to the library","response":500}
not ok 1350 [26-containersWait] POST containers/WaitTestingCtr/wait?condition=next-exit [-d {}]: expected curl to time out; it did not
#  expected: 
#    actual: 
not ok 1351 [26-containersWait] Received headers from /wait
#  expected: ~ .*HTTP.* 200 OK.*
#    actual: HTTP/1.1 500 Internal Server Error

not ok 1352 [26-containersWait] UNEXPECTED: curl on /wait returned results
#  expected: 
#    actual: 
not ok 1362 [26-containersWait] POST containers/waitNextExit/wait?condition=next-exit [-d {}] : status
#  expected: 200
#    actual: 500
#  response: {"cause":"unable to open a handle to the library","message":"unable to open a handle to the library","response":500}
not ok 1365 [27-containersEvents] GET libpod/events?stream=false&since=(T) : status
#  expected: 200
#    actual: 500
#  response: {"cause":"unable to open a handle to the library","message":"unable to open a handle to the library","response":500}
not ok 1369 [27-containersEvents] GET libpod/events?stream=false&since=(T) : status
#  expected: 200
#    actual: 500
#  response: {"cause":"unable to open a handle to the library","message":"unable to open a handle to the library","response":500}
not ok 1372 [27-containersEvents] GET events?stream=false&since=(T) : status
#  expected: 200
#    actual: 500
#  response: {"cause":"unable to open a handle to the library","message":"unable to open a handle to the library","response":500}
not ok 1376 [27-containersEvents] GET events?stream=false&since=(T)&type=remove : status
#  expected: 200
#    actual: 500
#  response: {"cause":"unable to open a handle to the library","message":"unable to open a handle to the library","response":500}
not ok 1379 [27-containersEvents] GET /v1.52/events?stream=false&since=(T) : status
#  expected: 200
#    actual: 500
#  response: {"cause":"unable to open a handle to the library","message":"unable to open a handle to the library","response":500}
not ok 1381 [27-containersEvents] GET /v1.52/events?stream=false&since=(T)&type=remove : status
#  expected: 200
#    actual: 500
#  response: {"cause":"unable to open a handle to the library","message":"unable to open a handle to the library","response":500}
not ok 1383 [27-containersEvents] GET events?stream=true: expected curl to time out; it did not
#  expected: 
#    actual: 
not ok 1384 [27-containersEvents] Received headers from /events
#  expected: ~ .*HTTP.* 200 OK.*
#    actual: HTTP/1.1 500 Internal Server Error

not ok 1885 [44-mounts] GET containers/b5af255d259b7673f0fbbd81c5ac88dad60f5e2630c9f8b4fde6b803ae293273/logs?stdout=true : status
#  expected: 200
#    actual: 500
#  response: {"cause":"unable to open a handle to the library","message":"failed to obtain logs for Container 'b5af255d259b7673f0fbbd81c5ac88dad60f5e2630c9f8b4fde6b803ae293273': unable to open a handle to the library","response":500}
not ok 1886 [44-mounts] 'df' output includes tmpfs name
#  expected: ~ .* /mytmpfs
#    actual: {"cause":"unable to open a handle to the library","message":"failed to obtain logs for Container 'b5af255d259b7673f0fbbd81c5ac88dad60f5e2630c9f8b4fde6b803ae293273': unable to open a handle to the library","response":500}
not ok 1894 [44-mounts] GET containers/ce9a4d101f99d2edcd2679d65e2305e720b0b706132fa1f2976bb69411f68b3f/logs?stdout=true : status
#  expected: 200
#    actual: 500
#  response: {"cause":"unable to open a handle to the library","message":"failed to obtain logs for Container 'ce9a4d101f99d2edcd2679d65e2305e720b0b706132fa1f2976bb69411f68b3f': unable to open a handle to the library","response":500}
not ok 1895 [44-mounts] compat mount subpath returns only selected subdir
#  expected: ~ .*hello1$
#    actual: {"cause":"unable to open a handle to the library","message":"failed to obtain logs for Container 'ce9a4d101f99d2edcd2679d65e2305e720b0b706132fa1f2976bb69411f68b3f': unable to open a handle to the library","response":500}

Seems to be errors during dlopen, obviously we cannot merge this when the tests fail. Unfortunately the lib does not log the error message from dlopen.

One thing I do not get why does this only fail as rootless but not root?

@vtushar06

vtushar06 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

@Luap99 dug at the rootless/root bit. same run, the two apiv2 jobs:

rootless   dlopen failures: 12   make: *** [localapiv2-bash] Error 17
root       dlopen failures: 0

and it started on a specific main run, both ran the full bash suite so its not a truncated one:

2026-08-10 05:18   plan 1..2182    0 failures
2026-08-10 15:43   plan 1..2188   12 failures

AUTOMATION_RELEASE is 20260722t094115z on both sides so not an image bump, and the only commit in between touches userns inspect and adds 6 apiv2 cases, nothing journald. so I couldnt pin it on either.

no idea on why root loads the same libsystemd fine. I only have the mac client here so I cant reproduce it.

agreed it cant merge while they fail, happy to leave it open or close and redo once they are sorted.

@Luap99

Luap99 commented Aug 17, 2026

Copy link
Copy Markdown
Member

agreed it cant merge while they fail, happy to leave it open or close and redo once they are sorted.

lets keep it open, it is important enough so we do not introduce new errors. I will try to see what is going on with dlopen here

@vtushar06 vtushar06 mentioned this pull request Aug 17, 2026
Open
@vtushar06

vtushar06 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

@Luap99 on the dlopen message - GetHandle never calls dlerror() at all, only GetSymbolPointer and Close do. Opened coreos/go-systemd#527:

old:  unable to open a handle to the library
new:  unable to open a handle to the library: /tmp/libbroken.so.0: invalid ELF header; libstrange.so: cannot open shared object file: No such file or directory

if you point a replace at that branch for one CI run it should say why libsystemd is not loading, which I could not work out from here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vtushar06 @Luap99